Sample deployment

The diagram below depicts a possible setup of replicated sites, followed by a description of individual elements present in the deployment. Options are then explained at large in future paragraphs.
Comments on the diagram above: 

• there are 3 sites: LON, NYC and SFO.

• in each site there is a running Infinispan cluster with a (potentially) different number of physical nodes: 3 nodes in LON, 4 nodes in NYC and 3 nodes in SFO

• the "users" cache is active in LON, NYC and SFO. Updates on the "users" cache in any of these sites gets replicated to the other sites as well

• it is possible to use different replication mechanisms between sites. E.g. One can configure SFO to backup data synchronously to NYC and asynchronously to LON

• the "users" cache can have a different configuration from one site to the other. E.g. it might be configured as distributed with numOwners=2 in the LON site, REPL in the NYC site and distributed with numOwners=1 in the SFO site

• JGroups is used for both inter-site and intra-site communication. RELAY2 is used for inter-site communication

• "orders" is a site local to LON, i.e. updates to the data in "orders" don't get replicated to the remote sites
  The following sections discuss specific aspects of cross site replication into more detail. The foundation of the cross-site replication functionality is RELAY2 so it highly recommended to read JGroups' RELAY2 documentation before moving on into cross-site.

  Configuration

  The cross-site replication configuration spreads over the following files:

1. the backup policy for each individual cache is defined in infinispan's .xml configuration file (infinispan.xml in attachment)

2. cluster's jgroups xml configuration file: RELAY2 protocol needs to be added to the JGroups protocol stack (jgroups.xml)

3. RELAY2 configuration file: RELAY2 has an own configuration file (relay2.xml)

4. the jgroups channel that is used by RELAY2 has its own configuration file (jgroups-relay2.xml)

   Infinispan XML configuration file

   The local site is defined in the the global configuration section. The local is the site where the node using this configuration file resides (in the example above local site is "LON").

   <global>
     ...
     <site local="LON"/>
     ...
   </global>

   The same setup can be achieved programatically:

   GlobalConfigurationBuilder lonGc = GlobalConfigurationBuilder.defaultClusteredBuilder();
   lonGc.site().localSite("LON");

   The names of the site (case sensitive) should match the name of a site as defined within JGroups' RELAY2 protocol configuration file.
   Besides the global configuration, each cache specifies its backup policy in the "site" element:

   <namedCache name="users">
      <sites>
        <backups>
          <backup site="NYC" backupFailurePolicy="WARN" strategy="SYNC" timeout="12000"/>
          <backup site="SFO" backupFailurePolicy="IGNORE" strategy="ASYNC"/>
          <backup site="LON" strategy="SYNC" enabled="false"/>
        </backups>
     </sites>
   </namedCache>

   The "users" cache backups its data to the "NYC" and "SFO" sites. Even though the "LON" appears as a backup site, it has the "enabled" attribute set to false so it will be ignored .        
   For each site backup, the following configuration attributes can be specified:

• strategy - the strategy used for backing up data, either "SYNC" or "ASYNC". Defaults to "ASYNC"

• backupFailurePolicy - Decides what the system would do in case of failure during backup. Possible values are:

  • IGNORE - allow the local operation/transaction to succeed

  • WARN - same as IGNORE but also logs an warning message. Default.

  • FAIL - only in effect if "strategy" is "SYNC" - fails local cluster operation/transaction by throwing an exception to the user

  • CUSTOM - user provided, see "failurePolicyClass" below

• failurePolicyClass - If the 'backupFailurePolicy' is set to 'CUSTOM' then this attribute is required and should contain the fully qualified name of a class implementing org.infinispan.xsite.CustomFailurePolicy

• timeout - The timeout(milliseconds) to be used when backing up data remotely. Defaults to 10000 (10 seconds)

The same setup can be achieved programatically:

ConfigurationBuilder lon = new ConfigurationBuilder();
lon.sites().addBackup()
      .site("NYC")
      .backupFailurePolicy(BackupFailurePolicy.WARN)
      .strategy(BackupConfiguration.BackupStrategy.SYNC)
      .replicationTimeout(12000)
      .sites().addInUseBackupSite("NYC")
    .sites().addBackup()
      .site("SFO")
      .backupFailurePolicy(BackupFailurePolicy.IGNORE)
      .strategy(BackupConfiguration.BackupStrategy.ASYNC)
      .sites().addInUseBackupSite("SFO")

The "users" cache above doesn't know on which cache on the remote sites its data is being replicated. By default the remote site writes the backup data to a cache having the same name as the originator, i.e. "users". This behaviour can be overridden with an "backupFor" element. For example the following configuration in SFO makes the "usersLONBackup" cache act as the backup cache for the "users" cache defined above in the LON site:

<infinispan>
  <namedCache name="usersLONBackup">
    <sites>
      <backupFor remoteCache="users" remoteSite="LON"/>
    </sites>
  </namedCache>
</infinispan>

The same setup can be achieved programatically:

ConfigurationBuilder cb = new ConfigurationBuilder();
cb.sites().backupFor().remoteCache("users").remoteSite("LON");

<infinispan>
..
  <global>
    <transport clusterName="infinispan-cluster">
      <properties>
        <property name="configurationFile" value="jgroups.xml"/>
      </properties>
    </transport>
  </global>
..
</infinispan>

Data replication

For both transactional and non-transactional caches, the backup calls are performed in parallel with local cluster calls, e.g. if we write data to node N1 in LON then replication to the local nodes N2 and N3 and remote backup sites SFO and NYC happen in parallel.

Taking a site offline

If backing up to a site fails for a certain number of times during an time interval, then it is possible to automatically mark that site as offline. When a site is marked as offline the local site won't try to backup data to it anymore. In order to be taken online a system administrator intervention being required.

<namedCache name="bestEffortBackup">
   ...
    <sites>
        <backups>
           <backup site="NYC" strategy="SYNC" backupFailurePolicy="FAIL">
               <takeOffline afterFailures="500" minTimeToWait="10000"/>
           </backup>
        </backups>
    </sites>
    ...
</namedCache>

lon.sites().addBackup()
      .site("NYC")
      .backupFailurePolicy(BackupFailurePolicy.FAIL)
      .strategy(BackupConfiguration.BackupStrategy.SYNC)
      .takeOffline()
         .afterFailures(500)
         .minTimeToWait(10000);

Reference

This document (Sept 2012) describes the technical design of cross site replication in more detail.